System and methods of adaptive relevancy prediction for autonomous driving

ABSTRACT

A method may include obtaining one or more inputs in which each of the inputs describes at least one of: a state of an autonomous vehicle (AV) or a state of an object; and identifying a prediction context of the AV based on the inputs. The method may also include determining a relevancy of each object of a plurality of objects to the AV in relation to the prediction context; and outputting a set of relevant objects based on the relevancy determination for each of the plurality of objects. Another method may include obtaining a set of objects designated as relevant to operation of an AV; selecting a trajectory prediction approach for a given object based on context of the AV and characteristics of the given object; predicting a trajectory of the given object using the selected trajectory prediction approach; and outputting the given object and the predicted trajectory.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Patent Application Ser. No.63/228,589, filed on Aug. 2, 2021, U.S. Patent Application Ser. No.63/228,590, filed on Aug. 2, 2021, and U.S. Patent Application Ser. No.63/366,738, filed on Jun. 21, 2022; the disclosures of which areincorporated herein by reference in their entireties.

The present disclosure generally relates to a system and methods ofadaptive relevancy prediction for autonomous driving and/or adaptivetrajectory prediction for autonomous driving.

BACKGROUND

An autonomous vehicle may scan an environment in which the autonomousvehicle is located using one or more sensors positioned on and/or aroundthe autonomous vehicle. The autonomous vehicle may detect objects basedon data collected by the sensors and avoid objects in a path of travelof the autonomous vehicle based on the detected objects.

The subject matter claimed in the present disclosure is not limited toembodiments that solve any disadvantages or that operate only inenvironments such as those described above. Rather, this background isonly provided to illustrate one example technology area where someembodiments described in the present disclosure may be practiced.

SUMMARY

One or more embodiments of the present disclosure may include a methodthat includes obtaining one or more inputs in which each of the inputsdescribes at least one of: a state of an autonomous vehicle (AV) or astate of an object; and identifying a prediction context of the AV basedon the inputs. The method may also include determining a relevancy ofeach object of a plurality of objects to the AV in relation to theprediction context; and outputting a set of relevant objects based onthe relevancy determination for each of the plurality of objects.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where identifying the prediction context may includeobtaining a first input of a localization of the AV in a map; obtaininga second input of a mode of operation of the AV; and obtaining a thirdinput of a driving strategy of the AV, and where the prediction contextis based on a combination of the localization, the mode of operation,and the driving strategy.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where determining the relevancy of a given objectmay include obtaining intent information and state information of thegiven object; following a tree construct based on the intentinformation, the state information, and the prediction context togenerate a relevancy value; and comparing the relevancy value to arelevancy threshold.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where following the tree construct may includedetermining whether a distance to the given object is beyond a thresholddistance; and based on the given object being beyond the thresholddistance, designating the given object as irrelevant.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the threshold distance varies based on theprediction context.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the method may also include classifying thegiven object as relevant based on the comparison to the relevancythreshold; and storing a decision at each node of the tree constructwith the given object in a cache.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the method may also include monitoringcharacteristics involved in the decisions at the nodes of the treeconstruct; based on a change in the characteristics, designating thegiven object as irrelevant; and removing the given object from thecache.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where determining the relevancy of a given objectmay include selecting one or more tree constructs to follow, where thetree constructs may be selected to correspond with the predictioncontext.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the method may also include obtaining intentinformation and state information of the given object; and wheredetermining the relevancy of a given object may include performing amulti-factorial analysis that considers multiple factors, at least oneof the factors related to the prediction context, the intentinformation, or the state information.

One or more embodiments of the present disclosure may include a methodthat may include obtaining a set of objects designated as relevant tooperation of an AV; and selecting a trajectory prediction approach for agiven object based on context of the AV and characteristics of the givenobject. The method may also include predicting a trajectory of the givenobject using the selected trajectory prediction approach; and outputtingthe given object and the predicted trajectory.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the method may also include selecting asecondary trajectory prediction approach for the given object; andpredicting a secondary trajectory of the given object using thesecondary trajectory prediction approach, where outputting the givenobject and the predicted trajectory may also include outputting a streamof the predicted trajectory and the secondary trajectory with the givenobject.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the secondary trajectory approach may bedifferent from the selected trajectory prediction approach and mayutilize more computing resources than the first trajectory predictionapproach.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the secondary trajectory approach may includeone of a learning-based trajectory prediction approach, astatistics-based trajectory prediction approach, or a multi-agent basedtrajectory prediction approach.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the method may also include monitoring acomputational intensity of a computing device performing the predictionof the trajectory of the given object.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the method may also include, based on thecomputational intensity exceeding a first threshold, decreasing at leastone of: (1) a length of time of the prediction of the trajectory, (2) anumber of points in the projected trajectory, or (3) a frequency atwhich the projected trajectory is updated.

One or more embodiments of the present disclosure may include a methodconsistent and/or compatible with one or more of the methods of thepresent disclosure, where the method may also include, based on thecomputational intensity being below a second threshold, increasing atleast one of: (1) a length of time of the prediction of the trajectory,(2) a number of points in the projected trajectory, or (3) a frequencyat which the projected trajectory is updated.

One or more embodiments of the present disclosure may includenon-transitory computer-readable medium storing instructions that, whenexecuted by one or more processors, are configured to cause a system toperform operations, such as operations consistent with one or more ofthe methods described herein and/or associated systems.

The object and advantages of the embodiments will be realized andachieved at least by the elements, features, and combinationsparticularly pointed out in the claims. It is to be understood that boththe foregoing general description and the following detailed descriptionare explanatory and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments will be described and explained with additionalspecificity and detail through the accompanying drawings in which:

FIG. 1 illustrates an example embodiment of an autonomous driving systemaccording to at least one embodiment of the present disclosure;

FIG. 2 illustrates an example system of adaptive prediction forautonomous driving according to one or more embodiments of the presentdisclosure;

FIG. 3 illustrates an example system of adaptive trajectory predictionfor autonomous driving according to one or more embodiments of thepresent disclosure;

FIG. 4 illustrates a flowchart of an example method of obtaining a setof relevant objects according to one or more embodiments of the presentdisclosure;

FIG. 5 illustrates a flowchart of another example method of obtaining aset of relevant objects according to one or more embodiments of thepresent disclosure;

FIG. 6 illustrates a flowchart of an example method of managing a cacheassociated with obtaining a set of relevant objects according to one ormore embodiments of the present disclosure;

FIG. 7 illustrates a flowchart of an example method of predictingtrajectories for relevant objects according to one or more embodimentsof the present disclosure;

FIG. 8 illustrates a flowchart of an example method of predictingtrajectories according to one or more embodiments of the presentdisclosure;

FIG. 9 illustrates a flowchart of an example method of monitoringcomputing resource usage according to one or more embodiments of thepresent disclosure; and

FIG. 10 is an example computing system.

DETAILED DESCRIPTION

Predicting the position and/or motion of objects in the vicinity of anautonomous vehicle (AV) during movement of the AV may require both fastand accurate predictions to prevent collisions and other accidents.Additionally, such trajectory predictions may require a certain degreeof granularity such that the predicted trajectories are useful fordecision-making by the AV. The present disclosure relates to, amongother things, predicting a relevancy and/or trajectory of objects in thevicinity of the AV such that the predicted trajectories may be outputquickly, accurately, and at a useful level of granularity. Trajectoryprediction according to the present disclosure may improve thedecision-making capabilities and overall safety of autonomous vehicles.

Embodiments of the present disclosure will be explained with referenceto the accompanying drawings.

Embodiments of the present disclosure will be explained with referenceto the accompanying drawings.

FIG. 1 illustrates an example embodiment of an autonomous driving system100 according to at least one embodiment of the present disclosure. Theautonomous driving system 100 may include one or more computer systemsand/or software modules configured to perform one or more operationsinvolved with driving an autonomous vehicle. Various aspects of theautonomous driving system 100 may be included on-board with theautonomous vehicle itself, or with a remote system in communication withthe autonomous vehicle, such as a cloud computing system, a server, or amobile device. In some embodiments, the autonomous driving system 100may include sensor drivers and a world engine 102 that is configured tocapture and process sensor data relating to an environment in which theautonomous vehicle travels, such as information relating to what objectsare present in the environment and where such objects are locatedrelative to the autonomous vehicle. The sensor data related to theautonomous vehicle's environment may be sent to a mapping system 104 togenerate a virtual map of the environment. In some embodiments, thevirtual map may be sent from the mapping system 104 back to the sensordrivers and world engine 102 to provide a map on which the sensor datarelating to environmental objects may be oriented. Additionally oralternatively, the virtual map may provide a map onto whichenvironmental topography may be plotted.

The virtual map of the mapping system 104 may be sent to a localizationsystem 106 and/or a routing system 108. The localization system 106 mayprovide an estimated location of the autonomous vehicle within theenvironment and/or the virtual map, and the routing system 108 maycompute a route between the estimated location of the autonomous vehicleto a designated destination along a valid path included in the virtualmap.

In some embodiments, the sensor data of the sensor drivers and worldengine 102 may be sent to a prediction system 110 that is configured topredict movement and/or trajectories of one or more objects in thevicinity of the autonomous vehicle and/or the autonomous vehicle itself.A decision system 112 may obtain the predicted object trajectories fromthe prediction system 110, and based on a location of the autonomousvehicle as determined by the localization system 106, the decisionsystem 112 may determine one or more driving decisions according tovarious driving rules. The driving decisions determined by the decisionsystem 112 may be sent to a planning system 114 that processes, parses,and/or queues the driving decisions for a downstream control system 116.In some embodiments, the control system 116 may generate control signalsthat are obtained by a drive-by-wire system 118 or another actuationsystem that controls one or more operations of the autonomous vehicle.

In some embodiments, the autonomous driving system 100 may include avisualization system 120 that is configured to obtain information fromthe sensor drivers and world engine 102, the prediction system 110, thedecision system 112, the planning system 114, the control system 116,the drive-by-wire system 118, or any other computer system of theautonomous driving system 100. The visualization system 120 may generatetwo-dimensional and/or three-dimensional models of the autonomousvehicle, objects in the vicinity of the autonomous vehicle, and/or theenvironment in which the autonomous vehicle operates. Additionally oralternatively, the visualization system 120 may provide a visualrepresentation of movement and/or predicted trajectories of theautonomous vehicle and/or any of the nearby objects.

In some embodiments, the autonomous driving system 100 may include adiagnostics system 122 that is configured to obtain information from thesensor drivers and world engine 102, the prediction system 110, thedecision system 112, the planning system 114, the control system 116,the drive-by-wire system 118, or any other computer system of theautonomous driving system 100. The diagnostics system 122 may rundiagnostic tests to assess the accuracy and/or validity ofdeterminations and decisions made by other computer systems of theautonomous driving system 100.

In these and other embodiments, the diagnostics system 122 may becommunicatively coupled to a simulations system 124 that provides avirtual environment in which the determinations and decisions made byother computer systems and/or software modules of the autonomous drivingsystem 100 may be simulated to identify any issues with the simulateddeterminations and decisions. Additionally or alternatively, thesimulations system 124 may obtain information from the drive-by-wiresystem 118 so that the simulations system 124 may run simulations basedon control signals implemented by the autonomous vehicle in the realworld. The simulations system 124 may virtually determine movements ofthe autonomous vehicle and/or of one or more objects in the vicinity ofthe autonomous vehicle. Based on the virtually determined movements, thesimulations system 124 may determine updated positions of the autonomousvehicle and/or of the surrounding objects. In some embodiments, thesimulations system 124 may provide the virtually determined movementsand/or the updated positions to the sensor drivers and world engine 102so that the total measurable world as represented by the sensor driversand world engine 102 may be updated based on the simulation results.

In some embodiments, the autonomous driving system 100 may include afleet management system 126 that obtains information from multipleautonomous vehicles communicatively coupled to the fleet managementsystem 126. The fleet management system 126 may relay informationbetween different autonomous vehicles in communication with the fleetmanagement system 126 to coordinate operations of the multipleautonomous vehicles. In these and other embodiments, a user managingfleet operations may use a mobile app 128 to receive information fromand/or send information or commands to a fleet of autonomous vehicles.

Modifications, additions, or omissions may be made to the autonomousdriving system 100 without departing from the scope of the presentdisclosure. For example, the designations of different elements in themanner described is meant to help explain concepts described herein andis not limiting. For instance, in some embodiments, the graphing module120 and the quadratic computation module 130 are delineated in thespecific manner described to help with explaining concepts describedherein but such delineation is not meant to be limiting. Further, theautonomous driving system 100 may include any number of other elementsor may be implemented within other systems or contexts than thosedescribed.

FIG. 2 illustrates an example system 200 of adaptive prediction forautonomous driving according to the present disclosure. In someembodiments, the adaptive prediction system may include an input manager210 that obtains one or more inputs that describe a state of the AVand/or a state of one or more objects, such as routing information 201,localization information 202, world engine updates 203, and/orhigh-definition mapping information 212, and generates a combined streamof inputs 215 based on the obtained inputs. In these and otherembodiments, a prediction configuration manager 205 may configure theinput manager 210, a prediction context manager 220, a prediction master(as illustrated in FIG. 3 ), and/or any other component of the adaptiveprediction system.

In some embodiments, the prediction configuration manager 205 mayprovide operational parameters of the various components of the system200. For example, properties related to available computing resources,receive user input directing aspects of the operation of the predictionsystem 200. For example, the prediction configuration manager 205 mayidentify an amount of computing resources to be used for variousoperations, a goal or mission of an AV, a size of a cache, varioustiming factors (e.g., time window of combining inputs, frequency ofgenerating predictions, among others), size of data to provide betweenand/or among components of the system 200, among other configurationparameters. In some embodiments, the prediction configuration manager205 may configure the handling of the one or more of the inputs includedin the combined stream of inputs 215 to align one or more properties ofthe inputs included in the combined stream of inputs 215. For example,the prediction configuration manager 205 may configure frequenciesand/or expected formatting of each of the inputs received by the inputmanager 210 such that the combined stream of inputs includes a singlefrequency of data capture and/or data format. As another example, theprediction configuration manager 205 may direct an input and/or outputtopology for the combined input stream 215, apply one or more heuristicrules to the combined input stream to configure the data included in thecombined input stream 215 for downstream analysis (e.g., by a relevancyprediction worker 230 and/or a trajectory prediction worker (asillustrated in FIG. 3 )).

In some embodiments, the input manager 210 may configure, manage, orotherwise process the input data based on configuration information ofthe prediction configuration manager 205. For example, the configurationinformation may be based on the environment in which the AV isoperating, a speed of the AV, a variability of the operation of the AVand/or the environment, etc. In these and other embodiments, theconfiguration information may be designated by an operator of the AV, adeveloper/programmer of the AV, any other user associated with the AV,automatically generated by the AV itself, etc., and/or combinationsthereof.

In some embodiments, the input manager 210 may combine the variousinputs, such as the routing information 201, the localizationinformation 202, and/or the world engine update information 203. In someembodiments, the input manager 210 may monitor a frequency of which thevarious inputs are received such that the combined stream of inputs 215may be aligned in frequency and/or packaged in a form such that theprediction context manager 220 is able to utilize the combined stream ofinputs 215. In some embodiments, the input manager 210 may discard someof the inputs that occur at a higher frequency and/or when the inputmanager 210 is configured to output information at a lower frequencysuch that the prediction context manager 220 receives a manageableamount of data. In these and other embodiments, the input manager 210may manage the size of the combined streams of data 215 to align withconfiguration information from the configuration manager 205 directing asize of the combined streams of data 215. Additionally or alternatively,the input manager 210 may remove duplicate information from the inputsto reduce the size of the combined streams of data 215.

In some embodiments, the input manager 210 may or may not perform somepreliminary processing of the inputs to offload some of the processingby the prediction context manager 220 and/or the relevancy predictionworkers 230. Such preliminary processing may include lightweightcomputing tasks such as averaging, regressions, formatting of datastructures, among others.

In some situations, computing systems included in the AV may detectmultiple objects as reflected in the inputs 201, 202, 203, and/or the HDmap 212, but only a subset of the objects may be relevant todecision-making by the AV. For example, people sitting at a table in arestaurant adjacent to a roadway may be irrelevant to thedecision-making of the AV. Without identifying the relevant subset ofobjects, a trajectory prediction system may functionally operate topredict the trajectory for each and every detected object, which may beprohibitively costly in terms of processing and computing resources. Insome embodiments, the prediction context manager 220 may determine oneor more pieces of contextual information relating to the AV that mayqualify a relevancy of objects included in the vicinity of the AV basedon the combined stream of inputs 215. For example, the predictioncontext manager 220 may identify an environment of the AV as beingwithin a warehouse. In a warehouse setting, most or all of the objectsincluded in the combined stream of inputs may be relevant because the AVmay be capable of moving to any location within the warehouseenvironment. As another example, the prediction context manager 220 mayidentify the environment of the AV as being a public highway road. In ahighway setting, not all of the objects included in the combined streamof inputs may be relevant because the AV may only be capable of movingon one half of the road and only in one lane at a given time.

In these and other embodiments, the prediction context manager 220 mayinclude a context cache 222 that stores contextual information about theAV (e.g., vehicle make/model, vehicle size, vehicle specifications,etc.) and/or the environment of the AV. The contextual informationstored in the context cache 222 may be retrieved more efficiently forfrequently used prediction operations. Additionally or alternatively,the cache 222 of the prediction context manager 220 may store a set ofdetected and/or processed objects. In this manner, the predictioncontext manager 220 may monitor for newly detected objects to determinetheir relevancy without the redundancy of having to cycle through and/orprocess each object detected.

In some embodiments, to determine the context, the prediction contextmanager 220 may utilize the combined stream of inputs 215. For example,the localization input 202 may indicate a particular location of the AVwithin the HD map 212. In some embodiments, the prediction contextmanager 220 may include multiple factors such as a physical location orother physical parameters, a mode of operation, and a mission/strategywhich may represent a purpose for operating the AV. For example, aphysical location of a warehouse, a highway, a rural road, or aresidential road changes the context. As another example, a speed atwhich the AV is traveling may impact which objects are relevant. As afurther example, when operating in a debugging mode, nearly every objectmay be considered as relevant while in a normal mode or an efficientmode of operation, fewer and fewer objects may be relevant. As anadditional example, a strategy of a leisurely drive may allow for a moreinclusive and robust number of objects as relevant while an aggressiveand urgent trip may utilize a lower number of relevant objects. In theseand other embodiments, the prediction context manager 220 may performany combination approach or multifactorial analysis to determine thecontext. For example, the prediction context manager 220 may utilize oneor more heuristics or rules that identify a given context based on themultiple factors. As another example, the prediction context manager 220may utilize a machine learning model to identify the context.

The contextual information determined by the prediction context manager220 may be used to generate a contextual object list 225 in which eachof the objects included in the combined stream of inputs 215 is pairedwith corresponding contextual information. In some embodiments, theobjects included in the contextual object list 225 may each include anintent and/or a state of the object. The state of the object mayrepresent physical parameters of the object, such as a distance betweenthe object and the AV, velocity, position, acceleration, size,orientation, among others. The intent of the object may includeinformation that may be gleaned based on other information regardingrelated to the object to facilitate a more accurate prediction oftrajectory. For example, the intent may include a classification of theobject (e.g., a pedestrian, another vehicle, a road hazard, a barrier, atree, among others), rules related to the object (e.g., a train willtravel on tracks, a mail truck or bus makes frequent stops, a bicyclecannot make instantaneous turns, among others), or other factors thatrelate to aspects of the objects other than physical parameters.

In some embodiments, the contextual object list 225 may be obtained bythe relevancy prediction workers 230 a, b, and c (which may be referredto collectively as relevancy prediction workers 230) and one or morerelevant objects 240 may be identified. In some embodiments, therelevancy prediction workers 230 may include a prediction treeconstructor 234 and/or a relevancy core 232 configured to classifyobjects included in the contextual object list as relevant or irrelevantobjects based on one or more decision trees determined by the predictiontree constructor 234 and/or other decision-making techniques.

The relevancy core 232 may be configured to determine the relevancy of agiven object. For example, the relevancy core 232 may apply a machinelearning process, a heuristic approach, a statistical analysis, or anyother methodology to designate the given object as relevant orirrelevant. For example, such an approach may utilize the state and/orintent relative to the context as inputs to the relevancy core 232 togenerate a relevancy score which may be compared to a threshold. Inthese and other embodiments, the intent and/or state of the given objectmay be analyzed based on the context to determine whether or not, withinthe identified context, the object according to the intent and/or stateare relevant to the operation of the AV. In some embodiments, therelevancy core 232 may utilize a random approach, including a “flip of acoin” to determine whether or not a given object is relevant.

The prediction tree constructor 234 may generate and/or include one ormore decision trees that facilitate classification of the objects asrelevant or irrelevant by applying logic of the decision trees accordingto the contextual information associated with each of the objects basedon the state and/or intent of the object. For example, a given decisiontree may begin by determining whether a given object is within athreshold distance to the AV in which the threshold distance may dependon contextual information associated with the given object and/or theAV. Responsive to determining the given object is further than thethreshold distance, the given decision tree may classify the givenobject as irrelevant. Responsive to determining the given object iswithin the threshold distance, the given decision tree may continue withone or more additional inquiries, such as determining whether the givenobject is within a region of interest of the AV, whether the givenobject is a pedestrian, and/or any other inquiries about the givenobject.

In some embodiments, based on a given object being classified asrelevant, the object and/or information associated with the given objectmay be added to a set of relevant objects 240. Such additionalinformation may include the intent information, the state information,and/or other information related to the given object. The set ofrelevant objects 240 may be output for downstream processing, such astrajectory prediction for the relevant objects, route prediction, amongothers.

In these and other embodiments, the relevancy prediction workers 230 mayinclude a relevancy cache 236 that stores object information relating toobjects included within a region of interest of the AV. In someembodiments, each object may conceptually include a corresponding cache.Caching the object information for objects included within the region ofinterest of the AV (e.g., a region within a threshold distance of theAV, such as the distance the AV is predicted to travel in the next 3seconds or 5 seconds, etc.) may facilitate more efficient computationsfor the objects including cached information because computationsrelated to such objects may be performed more frequently by therelevancy prediction workers 230. As such, computations related to suchcached object information may be performed more quickly, andcomputational resources may be focused on newly detected objects and/orobjects entering and/or exiting the region of interest of the AV. Forexample, if a given object is identified as relevant based on aparticular decision/prediction tree and/or operation of the relevancycore 232, the corresponding features and/or decisions identifying it asrelevant may be stored in the cache for that object. Based on one ofthose pieces of information changing, the information in the cache 236may be recalled and revisited in light of the updated information todetermine if the object has shifted to an irrelevant object (e.g., if anon-coming vehicle has passed the AV and is now driving further andfurther away from the AV). The cached information may also be used viceversa to determine if something has shifted from irrelevant to relevant.Additionally or alternatively, the cached information regarding therelevancy of the objects may be maintained to facilitate checking andverifying that the relevant objects are accounted for.

In some embodiments, the information used by the relevancy core 232and/or the prediction tree constructor 234 may be stored in the cache236 to facilitate rapid re-analysis of the relevancy of a given object.For example, for a given object being classified as relevant orirrelevant using a tree construct, the cache 236 may store the resultsat each branch of the tree construct and the piece of information usedto make that decision. As another example, the relevancy core 232 mayutilize a given machine learning approach and may store the inputs tothe machine learning algorithm and the result.

In some embodiments, the relevancy prediction performed by the relevancyprediction workers 230 may occur at a relatively high conceptual levelsuch that, with little computing resources, at least some of the objectsmay be discarded as irrelevant such that the objects classified asrelevant are reserved for the full, more-robust trajectory predictionanalysis (such as that described with reference to FIG. 3 ). Thetrajectory prediction analysis may be performed at a more detailed levelsuch that the path of travel of various objects may be facilitated.

Modifications, additions, or omissions may be made to the system 200without departing from the scope of the present disclosure. For example,the designations of different elements in the manner described is meantto help explain concepts described herein and is not limiting. Forinstance, in some embodiments, the prediction context manager 220, therelevancy prediction workers 230, and/or other components of the system200 are delineated in the specific manner described to help withexplaining concepts described herein but such delineation is not meantto be limiting. Further, the system 200 may include any number of otherelements or may be implemented within other systems or contexts thanthose described.

FIG. 3 illustrates an example system 300 of adaptive trajectoryprediction for autonomous driving, according to one or more embodimentsof the present disclosure. The system 300 may include a fine levelprediction adaptor 310, a prediction master 320, a master watchdog 325,one or more learning-based prediction adaptors 330, one or morestatistic-based prediction adaptors 340, one or more multi-agent basedprediction adaptors 350, one or more trajectory prediction workers 360.The fine level prediction adaptor 310 may receive a set of relevantobjects 305 and the prediction master 320 may select one or moretrajectory prediction workers 360 to be instantiated from thelearning-based prediction adaptors 330, statistic-based predictionadaptors 340, and/or multi-agent based prediction adaptors 350 togenerate one or more predicted trajectories of each of the relevantobjects 305 in the vicinity of an autonomous vehicle AV within a giventime period (e.g., within the next two seconds, three seconds, fiveseconds, etc.). In some embodiments, the set of relevant objects 305 mayinclude relevant objects identified by the relevancy prediction workers230 of FIG. 2 .

In some embodiments, the relevant objects 305 may first be obtained bythe fine level prediction adaptor 310 before being sent to theprediction master 320. The fine level prediction adaptor 310 mayconfigure one or more aspects of the relevant objects 305 such that theprediction master 320 may effectively analyze the relevant objects. Forexample, the fine level prediction adaptor 310 may reformat the relevantobjects and/or pair the relevant objects with suitable contextualinformation relating to a prediction type context 315 to provide theprediction master 320 with more information. Such context informationmay include the prediction context as determined by the predictioncontext manager 220 of FIG. 2 . In these and other embodiments, the finelevel prediction adaptor 310 may utilize context information (such asthat from the prediction context manager 220 of FIG. 2 ) to identify,direct, or control a size or duration of the prediction that theprediction master 320 may be causing to be generated. For example, basedon the context, the fine level prediction adaptor 310 may send aprediction type context 315 that directs the prediction manager togenerate one or more predicted trajectories for the next three seconds,five seconds, or any other temporal duration. The prediction typecontext 315 may additionally or alternatively identify a distance overwhich to predict the trajectory, a level of detail to which thetrajectory is to be predicted, or any other aspects of the trajectorythat may be advantageously adjusted based on the context in which thetrajectory is being predicted.

In some embodiments, the prediction master 320 may obtain the relevantobjects 305. In these and other embodiments, the prediction master 320may analyze the relevant objects 305 based on the prediction typecontext 315 associated with each of the relevant objects 305 todetermine one or more trajectory-prediction methods or approaches thatmay be beneficially used to predict the trajectory of each of therelevant objects 305. In these and other embodiments, the predictionmaster 320 may consider a given prediction type context includinginformation such as a location of the AV, a location of the relevantobjects, a vehicle type of the AV, and/or a type of the relevant objects(e.g., information indicating whether the relevant object is apedestrian, a wall, another vehicle, and/or any other objects that maybe identified in the surroundings of the AV) and select one or moresuitable trajectory-prediction methods or approaches according to theprediction type context 315. For example, the prediction master 320 mayselect a learning-based trajectory-prediction method for a given objectin response to determining that the given object is a pedestrian becausepedestrians are more likely to display unpredictable behaviors thatcannot be accurately modeled by statistical and/or heuristics-basedapproaches. As another example, a train may follow a regular andpredictable set of actions, lending such an object to prediction via astatistical and/or heuristic approach. In some embodiments, theprediction master 320 may select the learning-basedtrajectory-prediction method and/or a statistics-basedtrajectory-prediction method to predict the trajectories of the relevantobjects.

In some embodiments, the prediction master 320 may direct a givenprediction adaptor (such as the learning-based prediction adaptor 330,the statistic-based prediction adaptor 340, and/or the multi-agent basedprediction adaptor 350) to instantiate a given trajectory predictionworker 360 according to the trajectory prediction approach identifiedand/or various computational and/or trajectory parameters. For example,the prediction master 320 may direct the learning based predictionadaptor 330 to instantiate a trajectory prediction worker 360 using theRNN-based predictor 334 for a given object. In some embodiments, thetrajectory prediction worker 360 may include a prediction worker core362 that may be configured based on instructions provided by theprediction master 320. For example, the prediction worker core 362 mayperform the operations and processing tasks for the trajectoryprediction worker 360 to actually generate the prediction.

In some embodiments, the prediction master 320 may direct multipletrajectory prediction workers 360 be instantiated for any given relevantobject, such as the trajectory prediction workers 360 a-c. For example,the prediction master 320 may facilitate the determination of a primaryprediction and/or one or more secondary predictions. In selecting atrajectory prediction worker 360 to be instantiated, the predictionmaster 320 may select a reasonably consistent, reliable, and/or fastapproach as the primary prediction such that each object will have atleast one predicted trajectory as the primary prediction. One or more ofthe secondary predicted trajectories may represent a prediction that isgenerated in a more robust, more detailed, and/or more accurate mannerthan the primary prediction while taking longer and/or using morecomputational resources to generate the primary prediction. In someembodiments, the primary prediction may include a strict line predictionassumption model in which the object may be expected to move at the samevelocity and in the same single direction the object was previouslytraveling.

In some embodiments, the primary prediction may be determined accordingto a prevalent probabilistic behavior of a given object, and thesecondary predictions may be determined according to one or more otherprobabilistic behaviors of the given object. In some circumstances, theprimary prediction may include a high degree of confidence because theprevalent probabilistic behavior is highly prevalent relative to theother probabilistic behaviors. In these and other circumstances, thetrajectory prediction may be the same as or similar to the primaryprediction. In some circumstances, the primary prediction may include alow degree of confidence because one or more of the secondarypredictions may occur at a significant rate relative to the primaryprediction. In these and other circumstances, more than one trajectoryprediction may be outputted to account for the primary prediction andthe secondary predictions.

In some embodiments, the master watchdog 325 may be configured to adjustand/or update the settings or parameters via which the prediction master320 may instantiate the trajectory prediction workers 360. For example,the master watchdog 325 may identify an excessive use of CPU processes,memory, or other computing resources and may reduce the number ofsecondary predictions being generated.

In some embodiments, the adaptive prediction system 300 may include amaster watchdog 325 that is configured to regulate, budget, or otherwisecontrol the computations performed by the prediction master 320 and/orthe instantiated trajectory prediction workers 360. In some embodiments,the master watchdog 325 may adjust a computational speed, a number ofcomputation cycles, a duration of time within which computations are tobe completed, and/or other computational parameters (e.g., atrajectory-prediction window, a granularity of the predictions, and/or afrequency of prediction updates) of the prediction master 320 accordingto the computational burden caused by the prediction master 320 orimposed on the prediction master 320. In these and other embodiments,the computational burden on the prediction master 320 may be assessedaccording to an availability of computational resources, an average timetaken to perform computational tasks, and/or any other computationalperformance metrics monitored by the master watchdog 325.

In some circumstances, the master watchdog 325 may determine that thecomputational burden on the prediction master 320 and/or any associatedtrajectory prediction workers 360 are high and adjust the operations ofthe prediction master 320 accordingly. For example, the master watchdog325 may reduce a time period of trajectory prediction from five secondsto three seconds. As another example, the master watchdog 325 may reducea number of predicted trajectory points during the time period oftrajectory prediction from ten predicted points to four predictedpoints. As another example, the master watchdog 325 may reduce afrequency of prediction updating from 10 Hz to 5 Hz.

In other circumstances, the master watchdog 325 may determine that theprediction master 320 and/or any associated trajectory predictionworkers 360 are not computationally burdened and are capable ofperforming more and/or faster operations. In these and othercircumstances, the master watchdog 325 may increase the time period oftrajectory prediction, increase the number of predicted trajectorypoints, and/or increase the frequency of prediction updating.

In some embodiments, the master watchdog 325 may monitor for a baselineamount of rationality for the predicted trajectories. For example, themaster watchdog 325 may monitor for artifacts such as instantaneousdirection changes, teletransportation (instantaneously moving from oneposition to another position), travel at the speed of light, and/orother artifacts, and may flag or ignore the predictions that violate thebaseline rules.

In some embodiments, the learning-based trajectory prediction adaptor330 may include any of a number of learning-based trajectory predictionapproaches. For example, the learning-based trajectory predictionadaptor 330 may be facilitated by a multi-layer perceptron predictor(“MLP predictor”) 332, a recurrent neural network predictor (“RNNpredictor”) 334, and/or any other machine-learning (ML) based predictors336. In some embodiments, the prediction master 320 may determine whichlearning-based trajectory prediction approach is used to predict thetrajectories of the relevant objects. For example, the prediction master320 may select the RNN predictor 334 for predicting the trajectory of apedestrian demonstrating a meandering movement pattern and the MLPpredictor 332 for predicting the trajectories of pedestrians included ina crowd crossing a crosswalk. Additionally or alternatively, thelearning-based predictor to be used to predict the trajectories of therelevant objects may be selected based on one or more computingspecifications of the computing system on which the trajectoryprediction worker 360 is implemented or instantiated. Additionally oralternatively, the learning-based predictor may be selected according tothe contextual information associated with the AV (such as thatgenerated by the prediction context manager 220 of FIG. 2 ).

In some embodiments, the statistics-based trajectory-prediction adaptor340 may be facilitated by a rule-based predictor 342, aprobability-based predictor 344, and/or any other statistics-basedpredictors 346. The rule-based predictor 342 may predict the trajectoryof a given object that follows one or more rules guiding the movement ofthe given object. For example, a tractor in a farm field may be guidedby a rule indicating that the tractor should not leave the farm fieldand enter a connected public road. As another example, vehicles in awarehouse may be guided by specific movement routes within the warehouseaccording to a set of known rules. As an additional example, trains maybe presumed to remain on the train tracks and only travel on the traintracks as a rule. Additionally or alternatively, the rule-basedpredictor 342 may predict a trajectory of a given object based onsemantic information relating to the environment of the AV that providesrules for object movement (e.g., traffic signals, traffic signs, lanemarkings, etc.). For example, a given roadway may include one lane oftraffic in each direction and a curvature such that vehicles travelingtowards each other in opposite directions may appear to be on acollision course. However, the semantic information about the givenroadway including a central lane marking indicating different directionsof traffic flow and/or signage indicating road curvature may allow therule-based predictor to determine that the two oncoming vehicles willnot collide.

The probability-based predictor 344 may predict the trajectory of agiven object based on a likelihood that the given object will move in agiven trajectory according to previous movements of the given objectand/or similar objects. In some embodiments, the probability-basedpredictor 344 may be especially effective at predicting the trajectoryof a given object with a limited number of movement choices available tothe given object (e.g., at a traffic signal, on a highway ramp, in aroundabout, etc.). For example, a given delivery truck may typicallymake right turns during a delivery route. As another example, a givenforklift in a warehouse may be only able to make left turns. In theseand other examples, the probability-based predictor 344 may predict thetrajectory of the given delivery truck and the given forklift based onthe historic movement behavior of each of the vehicles.

In some embodiments, a multi-agent-based prediction adaptor 350 mayprovide information about movement of one or more objects included in amulti-agent group. In some embodiments, the multi-agent-based predictionadaptor 350 may include a large-scale simulation-based predictor 352, avehicle-to-everything (“V2X”) predictor 354, and/or any othermulti-agent predictors 356. The multi-agent predictors may each becommunicatively coupled to a given object included in the environment ofthe AV, and the multi-agent predictors may communicate trajectoryinformation to the AV such that trajectory information for each of thegiven objects is known. For example, a V2X predictor 354 coupled to alight pole around a corner of a public roadway relative to the AV mayprovide trajectory information about an oncoming vehicle that has notyet been detected by the AV.

In some embodiments, the large-scale simulation-based predictor 352 mayoperate in a similar manner to communications associated with the V2Xpredictor 354. For example, communication from the V2X predictor 354 maybe known to be completely accurate because it is communicated from theother object to inform the AV of the trajectory that the other objectwill be traveling. In a comparable manner, the large-scalesimulation-based predictor 352 may provide a known trajectory (thesimulation) that may be checked against the route the AV actually takessuch that the performance of the large-scale simulation can be checkedor verified. In some embodiments, a fleet may be managed, such as a hostof AVs operating in a warehouse. Continuing the example, each AV of thewarehouse may generate its own respective decisions for travel and maycommunicate the trajectory for the AV to a backend system and then on tothe other AVs of the fleet such that they can have a known expectedtrajectory for each of the AVs, rather than a prediction for the othervehicles in the fleet.

In some embodiments, the predicted trajectories as determined by each ofthe prediction approaches (learning-based, statistics-based, and/ormulti-agent-based) may be output as multiple streams of predictedtrajectories 365 in which each of the predicted trajectories correspondsto a respective relevant object as determined by a relevancy predictionworker (such as the relevancy prediction worker 230 of FIG. 2 ). In someembodiments, the streams of predicted trajectories 365 may include thepredicted trajectories (whether primary or secondary) for each of theplurality of relevant objects 305. Additionally or alternatively,multiple potential trajectories, which may include the likelihood of anyof the multiple potential trajectories, may be output in the multiplestreams 365.

In some embodiments, a prediction trajectory combiner 370 may combinethe multiple streams of trajectory predictions 365 as one or moretrajectory streams including the primary and/or secondary predictions380. In these and other embodiments, the prediction trajectory combiner370 may be configured to sort out the primary and/or secondarytrajectory predictions for a given object for a given point in time fromall of the trajectory predictions being generated by the trajectoryprediction workers 360, which may be output as the primary and/orsecondary predictions 380. In some embodiments, the primary and/orsecondary predictions 380 may be combined with information regarding therelevant objects 305, and may be provided to a down streamdecision-making engine which may decide where the AV is to go. Forexample, the decision-making engine may utilize the primary and/orsecondary predicted trajectories 380 and the relevant objects 305 tomake a decision of where to route the AV. In some embodiments, all ofthe primary and secondary predictions for a given relevant object at thegiven point in time may be combined in a single package for thedownstream systems, and/or the primary predictions and whichever of thesecondary predictions are completed in time for the relevant point intime.

Modifications, additions, or omissions may be made to the system 300without departing from the scope of the present disclosure. For example,the designations of different elements in the manner described is meantto help explain concepts described herein and is not limiting. Forinstance, in some embodiments, the prediction master 320, and/or thetrajectory prediction workers 360 are delineated in the specific mannerdescribed to help with explaining concepts described herein but suchdelineation is not meant to be limiting. Further, the system 300 mayinclude any number of other elements or may be implemented within othersystems or contexts than those described.

In some embodiments, various aspects of the system 200 and/or the system300 may be implemented to accomplish the operations described herein.For example, the input manager 210, the prediction context manager 220,the relevancy prediction workers 230, the fine level prediction adaptor310, the prediction master 320, the master watchdog 325, the learningbased prediction adaptors 330, the statistics based prediction adaptor340, the multi-agent based prediction adaptor 350, the trajectoryprediction worker 360, and/or the prediction trajectory combiner 370 mayinclude code and/or routines configured to enable a computing system toperform one or more operations. In these and other embodiments, theinput manager 210, the prediction context manager 220, the relevancyprediction workers 230, the fine level prediction adaptor 310, theprediction master 320, the master watchdog 325, the learning basedprediction adaptors 330, the statistics based prediction adaptor 340,the multi-agent based prediction adaptor 350, the trajectory predictionworker 360, and/or the prediction trajectory combiner 370 may beimplemented using hardware including a processor, a microprocessor(e.g., to perform or control performance of one or more operations), afield-programmable gate array (FPGA), or an application-specificintegrated circuit (ASIC). In some other instances, the input manager210, the prediction context manager 220, the relevancy predictionworkers 230, the fine level prediction adaptor 310, the predictionmaster 320, the master watchdog 325, the learning based predictionadaptors 330, the statistics based prediction adaptor 340, themulti-agent based prediction adaptor 350, the trajectory predictionworker 360, and/or the prediction trajectory combiner 370 may beimplemented using a combination of hardware and software. In the presentdisclosure, operations described as being performed by the input manager210, the prediction context manager 220, the relevancy predictionworkers 230, the fine level prediction adaptor 310, the predictionmaster 320, the master watchdog 325, the learning based predictionadaptors 330, the statistics based prediction adaptor 340, themulti-agent based prediction adaptor 350, the trajectory predictionworker 360, and/or the prediction trajectory combiner 370 may includeoperations that the respective components may direct a correspondingsystem to perform.

FIG. 4 is a flowchart of an example method 400 of obtaining a set ofrelevant objects according to at least one embodiment of the presentdisclosure. The method 400 may be performed by any suitable system,apparatus, or device. For example, the system 200 of FIG. 2 and/or thesystem of FIG. 3 and/or any of the subcomponents thereof may perform oneor more operations associated with the method 400. Although illustratedwith discrete blocks, the steps and operations associated with one ormore of the blocks of the method 400 may be divided into additionalblocks, combined into fewer blocks, or eliminated, depending on theparticular implementation.

At block 410, inputs describing a state of an AV may be obtained. Forexample, one or more inputs such as routing information, localizationinformation, world engine updates, and/or HD maps may be obtained. Suchinformation may be obtained by one or more sensors and/or one or moreother upstream systems.

At block 420, a prediction context of the AV may be identified based onthe inputs received at the block 410. For example, the context of the AVmay be predicted by a prediction context manager. Examples of such acontext may include operating in a warehouse, on a highway, the vehiclebeing parked, among others.

At block 430, a relevancy of each object for a set of objects may bedetermined based on the context and/or properties of the objects. Forexample, for a given object the relevancy of the object may bedetermined by a relevancy prediction worker. In these and otherembodiments, a prediction tree constructor and/or a relevancy core mayoperate to predict the relevancy of each object in the set of objects.

At block 440, a set of relevant objects may be output based on therelevancy of the objects. For example, as the various objects areidentified as relevant or irrelevant, the relevant objects may becombined into a set and may be output to other downstream systems, suchas the system 300 of FIG. 3 .

FIG. 5 illustrates a flowchart of an example method 500 of obtaining aset of relevant objects according to at least one embodiment of thepresent disclosure. The method 500 may be performed by any suitablesystem, apparatus, or device. For example, the system 200 of FIG. 2and/or the system 300 of FIG. 3 and/or any of the subcomponents thereofmay perform one or more operations associated with the method 500.Although illustrated with discrete blocks, the steps and operationsassociated with one or more of the blocks of the method 500 may bedivided into additional blocks, combined into fewer blocks, oreliminated, depending on the particular implementation.

At block 505, state information and/or intent information may beobtained for a set of objects. The intent information may includeinformation describing the purpose for the motion of the object and/orother information aside from pure physical parameters may be determined.For example, information describing the state of the AV may be obtainedbased on data from sensors and/or processed information based on thesensor data. As another example, intent information may be derived basedon a classification of the object. In some embodiments, the block 505may be similar or comparable to the block 410.

At block 510, the set of objects may be initialized such that the i^(th)object of the set of objects may be identified for analysis.

At block 515, a relevancy value may be determined for the i^(th) objectbased on the state information, the intent information, and/orprediction context information. For example, the state information, theintent information, and/or the prediction context information may beprovided to a relevancy prediction worker to predict the relevancy valueof the i^(th) object. For example, a machine learning process, aheuristic approach, or other analysis technique may be used to generatea value representative of the relative relevancy of the i^(th) object tothe operation of an AV.

At block 520, a determination may be made whether the relevancy value ofthe i^(th) object is above a threshold. If the relevancy value is notabove the threshold, the method 500 may proceed to the block 525. If therelevancy value is at or above the threshold, the method 500 may proceedto the block 530.

At block 525, a determination may be made whether there are additionalobjects in the set of objects to have their relevancy determined. Ifthere are additional objects, the method 500 may proceed to the block540. If there are no additional objects, the method 500 may proceed tothe block 545.

At block 530, the i^(th) object may be classified as relevant based onthe relevancy value being above the threshold.

At block 535, the i^(th) object may be added to a set of relevantobjects. For example, a set of relevant objects may be initialized as anull set and as each object is identified as relevant while analyzingthe various objects of the set of objects, the objects identified asrelevant may be added to the set of relevant objects. After the block535, the method 500 may proceed to the block 525 to analyze anyremaining objects in the set of objects for which relevancy has not yetbeen determined.

At block 540, the value of i may be incremented such that the nextobject in the set of objects may be analyzed. After the block 540, themethod 500 may proceed to the block 510 such that the next object in theset of objects may be analyzed to determine the relevancy of the nextobject.

At block 545, the set of relevant objects constructed by adding therelevant objects may be output. For example, the set of relevant objectsmay be passed downstream to other systems, such as a decision-makingsystem, a trajectory-prediction system, and/or any other relevantsystems related to operation of an AV.

FIG. 6 illustrates a flowchart of an example method 600 of managing acache associated with obtaining a set of relevant objects according toat least one embodiment of the present disclosure. The method 600 may beperformed by any suitable system, apparatus, or device. For example, thesystem 200 of FIG. 2 and/or the system 300 of FIG. 3 and/or any of thesubcomponents thereof may perform one or more operations associated withthe method 600. Although illustrated with discrete blocks, the steps andoperations associated with one or more of the blocks of the method 600may be divided into additional blocks, combined into fewer blocks, oreliminated, depending on the particular implementation.

At block 610, information related to a relevancy determination may bestored in a cache. For example, the decisions made at each branch in aprediction tree construct and/or the information used to make thedecision at the branches may be stored in the cache. As another example,the inputs to a machine-learning or heuristics-based approach todetermining relevancy may be stored in conjunction with the relevancydecision in the cache.

At block 620, one or more characteristics from the information relatedto the relevancy determination may be monitored. For example, thecharacteristics that formed the basis for the various decisions in theprediction tree construct that lead to a finding of relevance may bemonitored. As another example, the characteristics that were used as theinputs to the machine-learning approach to a relevancy determination maybe monitored.

At block 630, a determination may be made whether the characteristicshave changed beyond a threshold amount. For example, the values of thecharacteristics and/or the threshold(s) that lead to the variousdecisions may be monitored as the AV moves and/or progresses. In someembodiments, the determination may be made based on multiplecharacteristics and/or an entire tree structure. If the characteristicsare changed beyond the threshold(s), the method 600 may proceed to theblock 640. If the characteristics have not changed beyond thethreshold(s), the method 600 may return to the block 620 to continue tomonitor the characteristics of the relevancy determinations.

At block 640, the object may be designated as irrelevant. For example,based on the changes to the characteristics, the object may bereclassified as irrelevant instead of being relevant.

At block 650, the object may be removed from the cache. For example,after being classified as irrelevant, the object may be removed from thecache such that additional storage space is available in the cache. Forexample, the cache may store the relevant objects and/or informationrelated to the relevant objects. When no longer relevant, the objectsand/or their associated information may be removed from the cache. Theuse of the cache may facilitate rapid identification and monitoring ofobjects relevant to the operation of the AV.

FIG. 7 illustrates a flowchart of an example method 700 of predictingtrajectories for relevant objects according to at least one embodimentof the present disclosure. The method 700 may be performed by anysuitable system, apparatus, or device. For example, the system 200 ofFIG. 2 and/or the system 300 of FIG. 3 and/or any of the subcomponentsthereof may perform one or more operations associated with the method700. Although illustrated with discrete blocks, the steps and operationsassociated with one or more of the blocks of the method 700 may bedivided into additional blocks, combined into fewer blocks, oreliminated, depending on the particular implementation.

At block 710, a set of objects designated as relevant may be obtained.For example, the set of objects may be obtained following the methods400 and/or 500. Additionally or alternatively, a cloud-based system orother remote system may communicate the relevant set of objects to anAV.

At block 720, one or more trajectory prediction approaches may beselected for a given object based on context of the AV and/orcharacteristics of the given object. For example, a prediction mastermay identify which approach is to be used when predicting one or moretrajectories for the given object.

At block 730, a trajectory may be predicted using the selectedtrajectory prediction approach. For example, the prediction master mayinstantiate a trajectory prediction worker that is configured to use theprojection approach selected at the block 720. In some embodiments, ifmore than one approach is selected so that multiple trajectories arepredicted, multiple trajectory prediction workers may be instantiated,with each operating according to one particular approach and accordingto a particular set of parameters. In these and other embodiments, thepredicted trajectory may include a certain number of data points acrossa certain period of time (such as five data points over three seconds).

At block 740, the given object and the predicted trajectory may beoutput. For example, a data package that includes the given object andone or more of the predicted trajectories may be combined into a singledata package that may be provided to downstream systems, such as adecision-making system that may decide the manner in which the AV is tobe operated. In some embodiments, the block 740 may include combiningall relevant objects with their respective predicted trajectory ortrajectories.

FIG. 8 illustrates a flowchart of an example method of predictingtrajectories for relevant objects according to at least one embodimentof the present disclosure. The method 800 may be performed by anysuitable system, apparatus, or device. For example, the system 200 ofFIG. 2 and/or the system 300 of FIG. 3 and/or any of the subcomponentsthereof may perform one or more operations associated with the method800. Although illustrated with discrete blocks, the steps and operationsassociated with one or more of the blocks of the method 800 may bedivided into additional blocks, combined into fewer blocks, oreliminated, depending on the particular implementation.

At block 810, a set of objects designated as relevant may be obtained.The block 810 may be similar or comparable to the block 710.

At block 820, a first trajectory prediction approach may be selected fora given object, where the first trajectory prediction approach may beselected based on a context of the AV and characteristics of the givenobject. For example, a statistic-based prediction approach may beutilized when the characteristics and/or the context of the objectsuggest that the object is likely to follow one or more rules, such as atrain on a train track. As another example, a learning-based predictionapproach may be utilized when the characteristics and/or the context ofthe object suggest that the object is variable in how it will move, suchas a pedestrian crossing the street. In some embodiments, the block 820may be similar or comparable to the block 720.

At block 830, a first trajectory may be predicted for the given objectusing the selected first trajectory prediction approach. For example,the prediction master may instantiate a trajectory prediction workeraccording to the approach selected at the block 820. In someembodiments, the block 830 may be similar or comparable to the block730.

At block 840, a second trajectory prediction approach for the givenobject may be selected. In some embodiments, the first trajectoryprediction approach of the block 820 may be a reliable and consistenttrajectory prediction approach, such as a strict line predictionassumption model that may be used as a primary prediction. In these andother embodiments, the second trajectory prediction approach may includea more robust and/or more accurate prediction approach, while also usingadditional computing resources and/or taking more time than the firsttrajectory prediction approach.

At block 850, a determination may be made whether a timeout has elapsed.For example, the AV systems may expect an updated set of predictedtrajectories at a set frequency, such as 5 Hz. In these and otherembodiments, the timeout may be tied to the frequency such that at theconclusion of each period of time of the inverse of the frequency (e.g.,for a 5 Hz example every 20 ms), an updated set of relevant objects withassociated trajectories may be output. If the timeout has not elapsed,the method 800 may proceed to the block 860. If the timeout has elapsed,the method 800 may proceed to the block 870.

At block 860, a second trajectory may be predicted for the given objectusing the selected second trajectory prediction approach of the block840. For example, the prediction master may instantiate a trajectoryprediction worker according to the approach selected at the block 840.In these and other embodiments, the block 860 may represent a completionof the tasks or processes utilized to generate the second predictedtrajectory. After the block 860, the method 800 may proceed to the block880.

At block 870, the given object and the first predicted trajectory may beoutput. For example, in circumstances in which the second predictedtrajectory has not been completed when the timeout has elapsed, only thegiven object and the first predicted trajectory (e.g., the primarypredicted trajectory) may be outputted.

At block 880, the given object and both the first and the secondpredicted trajectories may be output. For example, in circumstances inwhich the second predicted trajectory has been completed when thetimeout has elapsed, the second predicted trajectory may be outputted inaddition to the first predicted trajectory for the object. the givenobject and the first predicted trajectory (e.g., the primary predictedtrajectory) may be outputted.

FIG. 9 illustrates a flowchart of an example method of monitoringcomputing resource usage according to at least one embodiment of thepresent disclosure. The method 900 may be performed by any suitablesystem, apparatus, or device. For example, the system 200 of FIG. 2and/or the system 300 of FIG. 3 and/or any of the subcomponents thereofmay perform one or more operations associated with the method 900.Although illustrated with discrete blocks, the steps and operationsassociated with one or more of the blocks of the method 900 may bedivided into additional blocks, combined into fewer blocks, oreliminated, depending on the particular implementation.

At block 910, the computational intensity of a computing deviceperforming trajectory predictions may be monitored. For example, amaster watchdog or other component or device may be configured tomonitor the computational intensity of the computing device thatperforms the trajectory predictions. The computational intensity may bebased off of a single characteristic of the computing device or multiplecharacteristics, including the number of processing cycles, an amount ofmemory being used, an amount of free storage, an amount of GPU cyclesbeing used, a temperature of the computing device, a temperature of theGPU, an amount of storage in a cache, among other monitorablecharacteristics of the computing device. In some embodiments, thecomputational intensity may include a value representative of theoverall load placed on the computing device.

At block 920, a determination may be made whether the computationalintensity has exceeded a first threshold. For example, the computationalintensity may be compared to the first threshold where the firstthreshold represents an excessive or high computational load on thecomputing device. In these and other embodiments, the first thresholdmay represent a point where the computing device may be damaged, or apoint where performance of the computing device is significantlydegraded, or any other metric. In some embodiments, the computationalintensity may be monitored for one or more of the trajectory predictionworkers specifically, rather than the entire computing device. Forexample, the monitoring of the block 910 and the determination of theblock 920 may be for the computational intensity for one or more of thetrajectory prediction workers. If the computational intensity hasexceeded the first threshold, the method 900 may proceed to the block930. If the computational intensity has not exceeded the threshold, themethod 900 may proceed to the block 940.

At block 930, the computing resource utilization may be decreased. Forexample, a duration of time for which a prediction is generated may bedecreased (e.g., from five seconds to three seconds), a number of pointsin the predicted trajectory may be decreased (e.g., from five points tothree points), a number of secondary predictions may be decreased (e.g.,from three secondary predictions to one or even zero secondaryprediction), a style of primary and/or secondary predictions may beadjusted (e.g., from a more computationally expensive predictionapproach to a less computationally expensive prediction approach), amongother changes. In some embodiments, this change may vary dynamically andbased on detected parameters of the computing system. After the block930, the method 900 may return to the block 910 to continue to monitorthe computational intensity of the computing device.

At block 940, a determination may be made whether the computationalintensity is below a second threshold. For example, the computationalintensity may be compared to the second threshold where the secondthreshold represents a low computational load on the computing device.In these and other embodiments, the second threshold may represent apoint where the computing resources are being wasted or areunderutilized, or any other metric. In some embodiments, thecomputational intensity may be monitored for one or more of thetrajectory prediction workers specifically, rather than the entirecomputing device. For example, the monitoring of the block 910 and thedetermination of the block 940 may be for the computational intensityfor one or more of the trajectory prediction workers. If thecomputational intensity his below the second threshold, the method 900may proceed to the block 950. If the computational intensity is notbelow the second threshold, the method 900 may return to the block 910.

At block 950, the computing resource utilization may be increased. Forexample, based on the availability of additional computing resources,more computing resources may be utilized in predicting the trajectories.For example, a duration of time for which a prediction is generated maybe increased (e.g., from two seconds to four seconds), a number ofpoints in the predicted trajectory may be increased (e.g., from twopoints to four points), a number of secondary predictions may beincreased (e.g., from one secondary prediction to three zero secondaryprediction), a style of primary and/or secondary predictions may beadjusted (e.g., from a less computationally expensive predictionapproach to a more computationally expensive prediction approach), amongother changes. After the block 950, the method 900 may return to theblock 910 to continue to monitor the computational intensity of thecomputing device.

Modifications, additions, or omissions may be made to the methods 400,500, 600, 700, 800, and/or 900 of FIGS. 4, 5, 6, 7, 8 , and/or 9,respectively, without departing from the scope of the disclosure. Forexample, the designations of different elements in the manner describedis meant to help explain concepts described herein and is not limiting.Further, the methods 400, 500, 600, 700, 800, and/or 900 may include anynumber of other elements or may be implemented within other systems orcontexts than those described. Additionally or alternatively, theoperations of the methods 400, 500, 600, 700, 800, and/or 900 may beperformed in different order, certain operations may be removed,expanded into additional operations, and/or performed simultaneously.

FIG. 10 illustrates an example computing system 1000, according to atleast one embodiment described in the present disclosure. The computingsystem 1000 may include a processor 1010, a memory 1020, a data storage1030, and/or a communication unit 1040, which all may be communicativelycoupled. Any or all of the adaptive prediction system 200 of FIG. 2and/or the system 300 of FIG. 3 may be implemented as a computing systemconsistent with the computing system 1000, including the predictioncontext manager 220, the relevancy prediction workers 230, theprediction master 320, and/or the trajectory prediction workers 360.

Generally, the processor 1010 may include any suitable special-purposeor general-purpose computer, computing entity, or processing deviceincluding various computer hardware or software modules and may beconfigured to execute instructions stored on any applicablecomputer-readable storage media. For example, the processor 1010 mayinclude a microprocessor, a microcontroller, a digital signal processor(DSP), an application-specific integrated circuit (ASIC), aField-Programmable Gate Array (FPGA), or any other digital or analogcircuitry configured to interpret and/or to execute program instructionsand/or to process data.

Although illustrated as a single processor in FIG. 10 , it is understoodthat the processor 1010 may include any number of processors distributedacross any number of network or physical locations that are configuredto perform individually or collectively any number of operationsdescribed in the present disclosure. In some embodiments, the processor1010 may interpret and/or execute program instructions and/or processdata stored in the memory 1020, the data storage 1030, or the memory1020 and the data storage 1030. In some embodiments, the processor 1010may fetch program instructions from the data storage 1030 and load theprogram instructions into the memory 1020.

After the program instructions are loaded into the memory 1020, theprocessor 1010 may execute the program instructions, such asinstructions to perform operations associated with prediction contextmanager 220, the relevancy prediction workers 230, the prediction master320, and/or the trajectory prediction workers 360.

The memory 1020 and the data storage 1030 may include computer-readablestorage media or one or more computer-readable storage mediums forhaving computer-executable instructions or data structures storedthereon. Such computer-readable storage media may be any available mediathat may be accessed by a general-purpose or special-purpose computer,such as the processor 1010. For example, the memory 1020 and/or the datastorage 1030 may store the contextual object list, the relevant objects,the prediction type context, and/or the predicted trajectories. In someembodiments, the computing system 1000 may or may not include either ofthe memory 1020 and the data storage 1030.

By way of example, and not limitation, such computer-readable storagemedia may include non-transitory computer-readable storage mediaincluding Random Access Memory (RAM), Read-Only Memory (ROM),Electrically Erasable Programmable Read-Only Memory (EEPROM), CompactDisc Read-Only Memory (CD-ROM) or other optical disk storage, magneticdisk storage or other magnetic storage devices, flash memory devices(e.g., solid state memory devices), or any other storage medium whichmay be used to store desired program code in the form ofcomputer-executable instructions or data structures and which may beaccessed by a general-purpose or special-purpose computer. Combinationsof the above may also be included within the scope of computer-readablestorage media. Computer-executable instructions may include, forexample, instructions and data configured to cause the processor 1010 toperform a certain operation or group of operations.

The communication unit 1040 may include any component, device, system,or combination thereof that is configured to transmit or receiveinformation over a network. In some embodiments, the communication unit1040 may communicate with other devices at other locations, the samelocation, or even other components within the same system. For example,the communication unit 1040 may include a modem, a network card(wireless or wired), an optical communication device, an infraredcommunication device, a wireless communication device (such as anantenna), and/or chipset (such as a Bluetooth device, an 802.6 device(e.g., Metropolitan Area Network (MAN)), a WiFi device, a WiMax device,cellular communication facilities, or others), and/or the like. Thecommunication unit 1040 may permit data to be exchanged with a networkand/or any other devices or systems described in the present disclosure.For example, the communication unit 1040 may allow the system 1000 tocommunicate with other systems, such as computing devices and/or othernetworks.

One skilled in the art, after reviewing this disclosure, may recognizethat modifications, additions, or omissions may be made to the system1000 without departing from the scope of the present disclosure. Forexample, the system 1000 may include more or fewer components than thoseexplicitly illustrated and described.

The embodiments described in the present disclosure may include the useof a special purpose or general-purpose computer including variouscomputer hardware or software modules. Further, embodiments described inthe present disclosure may be implemented using computer-readable mediafor having computer-executable instructions or data structures storedthereon.

Terms used in the present disclosure and especially in the appendedclaims (e.g., bodies of the appended claims) are generally intended as“open terms” (e.g., the term “including” should be interpreted as“including, but not limited to.”).

Additionally, if a specific number of an introduced claim recitation isintended, such an intent will be explicitly recited in the claim, and inthe absence of such recitation no such intent is present. For example,as an aid to understanding, the following appended claims may containusage of the introductory phrases “at least one” and “one or more” tointroduce claim recitations. However, the use of such phrases should notbe construed to imply that the introduction of a claim recitation by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim recitation to embodiments containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should be interpreted to mean “at least one”or “one or more”); the same holds true for the use of definite articlesused to introduce claim recitations.

In addition, even if a specific number of an introduced claim recitationis expressly recited, those skilled in the art will recognize that suchrecitation should be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, means at least two recitations, or two or more recitations).Furthermore, in those instances where a convention analogous to “atleast one of A, B, and C, etc.” or “one or more of A, B, and C, etc.” isused, in general such a construction is intended to include A alone, Balone, C alone, A and B together, A and C together, B and C together, orA, B, and C together, etc.

Further, any disjunctive word or phrase preceding two or morealternative terms, whether in the description, claims, or drawings,should be understood to contemplate the possibilities of including oneof the terms, either of the terms, or both of the terms. For example,the phrase “A or B” should be understood to include the possibilities of“A” or “B” or “A and B.”

All examples and conditional language recited in the present disclosureare intended for pedagogical objects to aid the reader in understandingthe present disclosure and the concepts contributed by the inventor tofurthering the art, and are to be construed as being without limitationto such specifically recited examples and conditions. Althoughembodiments of the present disclosure have been described in detail,various changes, substitutions, and alterations could be made heretowithout departing from the spirit and scope of the present disclosure.

What is claimed is:
 1. A method comprising: obtaining one or more inputsin which each of the inputs describes at least one of: a state of anautonomous vehicle (AV) or a state of an object; identifying aprediction context of the AV based on the inputs; determining arelevancy of each object of a plurality of objects to the AV in relationto the prediction context; and outputting a set of relevant objectsbased on the relevancy determination for each of the plurality ofobjects.
 2. The method of claim 1, wherein identifying the predictioncontext comprises: obtaining a first input of a localization of the AVin a map; obtaining a second input of a mode of operation of the AV; andobtaining a third input of a driving strategy of the AV, and wherein theprediction context is based on a combination of the localization, themode of operation, and the driving strategy.
 3. The method of claim 1,wherein determining the relevancy of a given object comprises: obtainingintent information and state information of the given object; followinga tree construct based on the intent information, the state information,and the prediction context to generate a relevancy value; and comparingthe relevancy value to a relevancy threshold.
 4. The method of claim 3,wherein following the tree construct includes: determining whether adistance to the given object is beyond a threshold distance; and basedon the given object being beyond the threshold distance, designating thegiven object as irrelevant.
 5. The method of claim 4, wherein thethreshold distance varies based on the prediction context.
 6. The methodof claim 3, further comprising: classifying the given object as relevantbased on the comparison to the relevancy threshold; and storing adecision at each node of the tree construct with the given object in acache.
 7. The method of claim 6, further comprising: monitoringcharacteristics involved in the decisions at the nodes of the treeconstruct; based on a change in the characteristics, designating thegiven object as irrelevant; and removing the given object from thecache.
 8. The method of claim 1, wherein determining the relevancy of agiven object comprises selecting one or more tree constructs to follow,the tree constructs selected to correspond with the prediction context.9. The method of claim 1, further comprising: obtaining intentinformation and state information of the given object; whereindetermining the relevancy of a given object comprises performing amulti-factorial analysis that considers a plurality of factors, at leastone of the plurality of factors related to the prediction context, theintent information, or the state information.
 10. One or morenon-transitory computer-readable media containing instructions which,when executed by one or more processors, cause a system to performoperations, the operations comprising: obtaining one or more inputs inwhich each of the inputs describes at least one of: a state of anautonomous vehicle (AV) or a state of an object; identifying aprediction context of the AV based on the inputs; determining arelevancy of each object of a plurality of objects to the AV in relationto the prediction context; and outputting a set of relevant objectsbased on the relevancy determination for each of the plurality ofobjects.
 11. The non-transitory computer-readable media of claim 10,wherein identifying the prediction context comprises: obtaining a firstinput of a localization of the AV in a map; obtaining a second input ofa mode of operation of the AV; and obtaining a third input of a drivingstrategy of the AV, and wherein the prediction context is based on acombination of the localization, the mode of operation, and the drivingstrategy.
 12. The non-transitory computer-readable media of claim 10,wherein determining the relevancy of a given object comprises: obtainingintent information and state information of the given object; followinga tree construct based on the intent information, the state information,and the prediction context to generate a relevancy value; and comparingthe relevancy value to a relevancy threshold.
 13. The non-transitorycomputer-readable media of claim 12, wherein following the treeconstruct includes: determining whether a distance to the given objectis beyond a threshold distance; and based on the given object beingbeyond the threshold distance, designating the given object asirrelevant.
 14. The non-transitory computer-readable media of claim 13wherein the threshold distance varies based on the prediction context.15. The non-transitory computer-readable media of claim 12, theoperations further comprising: classifying the given object as relevantbased on the comparison to the relevancy threshold; and storing adecision at each node of the tree construct with the given object in acache.
 16. The non-transitory computer-readable media of claim 15, theoperations further comprising: monitoring characteristics involved inthe decisions at the nodes of the tree construct; based on a change inthe characteristics, designating the given object as irrelevant; andremoving the given object from the cache.
 17. The non-transitorycomputer-readable media of claim 10, wherein determining the relevancyof a given object comprises selecting one or more tree constructs tofollow, the tree constructs selected to correspond with the predictioncontext.
 18. A system comprising: one or more processors one or morenon-transitory computer-readable media containing instructions which,when executed by the one or more processors, cause the system to performoperations, the operations comprising: obtaining one or more inputs inwhich each of the inputs describes at least one of: a state of anautonomous vehicle (AV) or a state of an object; identifying aprediction context of the AV based on the inputs; determining arelevancy of each object of a plurality of objects to the AV in relationto the prediction context; and outputting a set of relevant objectsbased on the relevancy determination for each of the plurality ofobjects.
 19. The non-transitory computer-readable media of claim 10,wherein identifying the prediction context comprises: obtaining a firstinput of a localization of the AV in a map; obtaining a second input ofa mode of operation of the AV; and obtaining a third input of a drivingstrategy of the AV, and wherein the prediction context is based on acombination of the localization, the mode of operation, and the drivingstrategy.
 20. The non-transitory computer-readable media of claim 10,wherein determining the relevancy of a given object comprises: obtainingintent information and state information of the given object; followinga tree construct based on the intent information, the state information,and the prediction context to generate a relevancy value; and comparingthe relevancy value to a relevancy threshold.